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ABSTRACT 

This monograph contains the instruments designed or modified 
by the research staff of the Urban Systemic Initiative (USI) during the 
course of the evaluation research grant from the National Science Foundation 
(NSF) . Because the project was designed to determine the impact of USI reform 
in four cities and to test the NSF Six-Driver model for reform, it was 
necessary to create instruments that would elicit information to meet both 
research goals. The document is organized into three sections corresponding 
to the three studies that constitute the scope and focus of the research. 
There are three studies (the Math and Science Attainment Study, the Study of 
the Enacted Curriculum, and the Policy Study) , but the Policy Study is made 
up of multiple studies related to policy. All of the instruments in this 
collection were designed to measure aspects of the NSF' s six project drivers 
or to support the collection of data related to the drivers (e.g., cover 
letters for studies) . Instruments, which include observation protocols, 
surveys, interview protocols, and focus group guides, are grouped into: (1) 

informed consent letters and documents; (2) statements about confidentiality 
and protection of participants; (3) the Mathematics/Science Attainment Study; 
(4) the Study of the Enacted Curriculum; and (5) the various aspects of the 
Policy Study. (Contains 10 references.) (SLD) 
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This monograph contains the instruments designed or modified by the USI research staflf 
during the course of the evaluation research grant fiom the National Science Foundation. 

Because the project was designed both to determine the impact of USI reform in four cities 
and to test the NSF Six- Driver model for reform, it was necessary to create instmments that 
would elicit information to meet both research goals. 

This document is organized in three sections corresponding to the three studies that constitute 
the scope and focus of our research. Although there are three studies, the Math and Science 
Attainment Study; the Study of the Enacted Curriculum; and the Policy Study, the pohcy 
study is actually multiple studies that are all related to policy (i.e. Principals, District 
Administrators, Professional Development, Stakeholder Mobilization, and the Community 
Context). All of the instruments were designed to measure aspects of NSF’s six drivers: 

Driver 1 . Implementation of comprehensive, standards-based curricula as represented in instructional 
practice, including student assessment, in every classroom, laboratory, and other learning experience 
provided through the system and its partners. 

Driver 2. Development of a coherent, consistent set of policies that support: provision of high quality 
mathematics and science education for each student; excellent preparation, continuing education, and 
support for each mathematics and science teacher (including all elementary teachers); and administrative 
support for all persons who work to dramatically improve achievement among all students served by the 
system. 

Driver 3. Convergence of the usage of all resources that are designed for or that reasonably could be used to 
support science and mathematics education-- fiscal, intellectual, material, curricular, and extra-curricular— 
into a focused and unitary program to constantly upgrade, renew, and improve the educational program in 
mathematics and science for all students. 

Driver 4. Broad-based support from parents, policymakers, institutions of higher education, business and 
industry, foundations, and other segments of the community for the goals and collective value of the 
program, based on rich presentations of the ideas behind the program, the evidence gathered about its 
successes and its failures, and critical discussions of its efforts. 

Driver 5. Accumulation of a broad and deep array of evidence that the program is enhancing student 
achievement, through a set of indices that might include achievement test scores, higher level courses 
passed, college admission rates, college majors. Advanced Placement Tests taken, portfolio assessment, and 
ratings from summer employers, and that demonstrate that students are generally achieving at a significantly 
higher level in science and mathematics. 

Driver 6. Improvement in the achievement of all students, including those historically underserved. 

Dr. M. Yvette Baber and Bridget McCourt have done the research community a tremendous 
service by organizing this monograph. We all hope that these instruments will be modified 
and used by researchers who share a commitment to strengthening our public schools. 

Kathryn M. Borman, Principal Investigator 
Gladis Kersaint, Co- Principal Investigator 
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This study received approval from the University of South Florida Division of Research 
Compliance in June 1999. In fall 1999 and spring 2000, teachers received letters inforrning 
them of the purposes of the study and confidentiality procedures. They were asked to 
voluntarily participate in the research. Teachers gave signed consent for classroom 
observations, focus groups, and survey responses. 



In the spring 2000 wave of research, parents received a separate informed consent letter for 
the student survey portion of the Study of the Enacted Curriculum. This letter was also 
translated into Spanish. 



Also in spring 2000, faculty in the participating schools received an informed consent form 
for the School Culture Quality Survey (SCQS). District administrators were also sent an 
informed consent letter requesting their participation in an interview and the SCQS. 



During the fall 2000 and spring 2001 research, additional consent forms were approved for 
student participation in the Student Engagement Study. These documents were translated into 
Spanish and Haitian Creole to accommodate the linguistic needs of parents and guardians. 



Respondents in the fall 2000 and spring 2001 Study of the Community Context received a 
one- sheet explanation of the research and their confidentiality rights. They provided signed 
consent at the top of the interview protocol at the time of the interview. 



Teacher Consent Form - Fall 1999 
USF Institutional Review Board-Behavioral/Social 
Consent to Participate in a Research Study 

Title of Study: Assessing the Impact of the National Science Foundation's Urban Systemic Initiative 

Principal Investigator Name; Kathryn M. Borman Telephone # 974-9058 

Department David C. Anchin Center Mail-point EDU 162 Tel # 974-5959 

Other Research Staff: Susan D. Greenbaum, Ph.D; Gladis Kersaint, Ph.D. ;Edward J. Glickman, 
Ph.D.;Sandra L. Cade; M.A.T.; Graduate Research Associates [Ellen Puccia, Linda Callejas, Cathleen 
Larrimore, Bridget McCourt, Danielle O'Connor, Jessica Pearlman, Christiana Schumann, Julian 
Smothers] 

Study Location(s): Chicago, IL; El Paso, TX; Memphis, TN; Miami, FL. 

Purpose of Study 

The purpose of this research study is to determine the impact of USI reforms in four sites: Chicago, El 
Paso, Memphis, and Miami. The study will determine how reforms in curriculum and instruction 
affect teacher and student outcomes at the classroom level (with standards -based curriculum in math 
and science). The study will also investigate the impact of USI reform on policy, assessment, 
standards, and relationships with community and businesses. 

Duration and Location of the Study 

Your participation in this study will last for approximately 1 - 6 hours and will take place at sites in 
Chicago El Paso Memphis Miami 

The number of other people that might participate in this study is: 7200 school aged children and 700 
adults. 



Procedures 

You understand that during this study, the following procedures will occur. 

You will either: 1) be interviewed by a project researcher, 2) complete a survey related to the goals of 
the course, or 3) be part of a classroom observation 

Potential Risks 

You understand that there are no anticipated risks associated with the study from your participation in 
this study. 

Benefits 

You understand that there will be no direct benefits of your participation in this study, but there may 
be possible benefits to others that may include findings that contribute to the improvement of 
instruction and professional development at the school-site and district level. The results of the 
research should also be applicable to other USI sites in the United States. 

Confidentiality 

The confidentiality of the records shall be maintained unless otherwise required by law. 

Confidentiality of records will be maintained by assigning pseudonyms to interviewees, assigning 
codes to tape-recorded interviews and their transcripts. Only USI research project staff will have 
access to the data. The project will secure all files in the David C. Anchin Center (College of 
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Education). At the end of the project, tapes will be erased and non-essential files will be destroyed in 
order to protect the confidentiality of research subjects. 

Compensation for Participation 

You will not be paid for participation in this study. 

Volunteering to Be Part of this Research Study 

You understand that participation in this study is voluntary. You understand that you may withdraw 
from the study at any time without penalty or loss of services, to which you are otherwise entitled 
You also understand that the investigator has the right to remove you from the study at any time. 

Questions and Contacts 

If you have any questions about this research study, you may contact Dr. Kathryn Borman at the 
University of South Florida in Tampa (813-974-9058) . If you have any questions about your rights 
as a person taking part in a research study, you may contact a member of the Division of Compliance 
Services at the University of South Florida at (813) 631-4498. 

Your Consent — By signing this form I agree that: 

• 1 have fully read or have had read and explained to me in my native language this informed consent 
form describing a research project. 

• 1 have had the opportunity to question one of the persons in charge of this research and have received 
satisfactory answers. 

• 1 understand that 1 am being asked to participate in research. 1 understand the risks and benefits, and I 
freely give my consent to participate in the research project outlined in this form, under the conditions 
indicated in it. 

• 1 have been given a signed copy of this informed consent form, which is mine to keep. 



Signature of Participant Printed Name of Participant 



Date 



Signature of Witness Printed Name of Witness Date 

Investigator Statement 

1 have carefully explained to the subject the nature of the above protocol. 1 hereby certify that to the best 
of my knowledge the subject signing this consent form understands the nature, demands, risks and 
benefits involved in participating in this study. 



Signature of Investigator Printed Name of Investigator 



Date 



Institutional Approval of Study and Informed Consent 

This research project/study and informed consent form were reviewed and approved by the University of 
South Florida Institutional Review Board for the protection of human subjects. This approval is valid until 
the date provided below. The board may be contacted at (813) 631-4498. 
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PARENTAL INFORMED CONSENT - STUDY OF THE ENACTED CURRICULUM 



Dear Parent: 

Your child’s teacher is participating in research Assessing the Impact of the National Science 
Foundation 's Urban Systemic Initiative conducted by the Anchin Center of the University of South 
Florida. The National Science Foundation has implemented Urban Systemic Initiatives in 20 cities to 
increase student achievement and reduce achievement differences for traditionally underserved 
students in mathematics, science, and technology. Our project aims at discovering similarities and 
differences among school sites in four cities and examining the extent to which the National Science 
Foundation’s formula for systemic reform can be made enduring. An important part of the study is to 
determine how reforms in mathematics and science influence teacher and student outcomes at the 
classroom level. 

A researcher from the University of South Florida will ask all of the students in your child’s 
classroom to voluntarily complete a survey about instructional activities during mathematics or 
science instruction. This survey usually takes about 30 minutes during class time to complete. Survey 
topics include student views on homework, problem-solving activities, group work, hands-on 
materials, and technology. In a several classrooms, a few students may also be interviewed about these 
topics. These audio -recorded interviews usually take 15 to 20 minutes. We will always make 
arrangements in advance with the school district office, the principal, and the teacher to ensure that 
these activities minimally interfere with the regular school day. 

Your child’s participation is voluntary. Your child is not required to complete the survey or the 
interview and there will be no penalty or other consequence if you choose not to allow your child to 
participate. All participants will remain anonymous. To insure anonymity we will not ask for 
children’s names and we will not identify the teacher or the school by name. All tapes will be erased at 
the end of the study. We know of no negative outcomes from your child’s responding to such items 
and your child may benefit by telling us directly about their school experiences. 

If you have additional questions about this study please contact your child’s teacher. This 
study has been approved by the Institutional Review Board of the University of South Florida and is 
monitored by the Office of Compliance Services. You may contact them at 813-974-5638 and you 
may also contact Dr. Kathryn Borman at the University of South Florida in Tampa at 813-974-9058. 
Your son or daughter’s participation is greatly appreciated. 

If you do not wish for your child to participate, please have your child return this form with 
your signature below: 

I DO NOT WISH MY CHILD TO PARTICIPATE IN THIS STUDY EVALUATING THE EFFECTIVENESS 
OF THE USI GRANT. 



Signature of Chiid 



Printed Name of Child 



Date 



Signature of Parent 



Printed Name of Parent 



Date 
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FACULTY INFORMED CONSENT- SCHOOL CULTURE QUALITY SURVEY 

Dear (Teacher Name): 

This letter is asking for your permission to complete a survey for research Assessing the 
Impact of the National Science Foundation 's Urban Systemic Initiative conducted by the David C. 
Anchin Center at the University of South Florida. An important part of the research ^udy is to [ 
determine how reform efforts have affected the cultural climate in schools that participated in the 
National Science Foundation's Urban Systemic Initiative grant. 

We are asking you to complete a survey about your perceptions of the climate in your school. 
The School Culture Quality Survey is an instrument designed to assist in understanding the culture of | 
schools. The instrument contains 36 scored items that are divided into four scales, Shared Vision, 
Facilitative Leadership, Teamwork, and the Learning Community. Responses will be analyzed and a I 
summary report utilizing charts, graphs, and summarized data that will give a picture of the! 
perceptions of the cultural climate of the school. 

This survey usually takes about 30 minutes to complete. Your participation in this research is 
voluntary and your refusal to participate will not involve any penalty. We do not anticipate any risks 
resulting from your participation in the study. We will preserve the confidentiality of your 
participation in the study by not identifying you, other staff members, or your school by name. Your [ 
school will receive a copy of the results of the survey and general results will be reported to the 
National Science Foundation. 

If you have any questions about your participation in this research study you may ask the 
researcher administering the survey, you may contact Dr. Kathryn Borman at the University of South I 
Florida in Tampa at 813-974-9058, and the Office of Compliance Services at 813-974-5638. 



Grant # 9874246 

School Staff Informed Consent Form 

By signing this form I agree that: I have fully read and had explained to me this informed consent form | 
describing this research project. 

I have had the opportunity to question one of the persons in charge of this research and have received 
satisfactory answers. 

I understand that I am being asked to participate in research. I understand the risks and benefits, and I freely give I 
my consent to participate in the research project outlined in this form, under the conditions indicated in it. 

I have been given a signed copy of this informed consent form, which is mine to keep. 



Signature of Participant Printed Name of Participant 



Date 



Signature of Researcher Printed Name of Researcher 



Date 



DISTRICT INTERVIEW/ SCHOOL CULTURE QUALITY SURVEY 
INFORMED CONSENT 



Dear (District staff): 



Your district is participating in research entitled Assessing the Impact of the National Science 
Foundation 's Urban Systemic Initiative conducted by the David C. Anchin Center of the University of 
South Florida. The National Science Foundation has implemented Urban Systemic Initiatives (USl) in 
20 cities to increase student achievement and reduce achievement differences for traditionally 
underserved students in mathematics, science, and technology. Our project aims at evaluating the 
results of these initiatives, discovering similarities and differences among school sites in four cities 
and examining the extent to which the National Science Foundation’s formula for systemic reform can 
be made enduring. An important part of the research study is to determine how reform efforts have 
affected the cultural climate in the district that participated in the National Science Foundation’s 
Urban Systemic Initiative grant. 

We hope you will contribute to this multi-city study by participating in an interview about 
your perceptions of the district’s USl and completing a survey about your perceptions of the climate in 
your district workplace. Research on school reform reveals that teacher perceptions about the climate 
and collegial relationships in their schools are related to student learning. The School Climate Quality 
Survey is an instrument designed to assist in understanding the culture of schools impacts reform 
initiatives. The instrument contains 36 scored items that are divided into four scales. Shared Vision, 
Facilitative Leadership, Teamwork, and the Learning Community. Responses will be analyzed and a 
report utilizing charts, graphs, and summarized data that will give a picture of the perceptions of the 
cultural climate of the school. The survey usually takes about 15 to 30 minutes to complete. 

Your participation in this research is voluntary and your refusal to participate will not involve 
any penalty. All participants will remain anonymous. We will preserve the confidentiality of your 
participation by not identifying you, other staff members, or schools by name. We know of no 
negative outcomes from your responding to such items and you may benefit by telling us directly 
about your workplace climate and experiences with the USl grant. All audio-recorded tapes will be 
erased after they are transcribed without using names. Your district will receive a copy of the 
aggregated results of the survey and general results will be reported to the National Science 
Foundation. 

This study has been approved by your school district and by the Institutional Review Board of the 
University of South Florida and is monitored by the Office of Compliance Services. You may contact 
them at 813-974-5638 and you may also contact Dr. Kathryn Borman at the University of South 
Florida in Tampa at 813-974-9058. Your assistance in this project is greatly appreciated. 
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Parent / Child Consent Form for Student Engagement Study 



Dear Parent: 

We are asking your permission for your child to participate in a research study to determine 
the impact of the National Science Foundation’s Urban Systemic Initiative reforms in mathematics, 
science, and technology in four sites, Chicago, El Paso, Memphis, and Miami. The Student 
Engagement Study wants to find out how students think and feel about the math and science reforms 
that are taking place in their classrooms. We will also be contacting some of you (the parents) to talk 
with you about your impressions of the math and science reforms being enacted in your school district. 

For the student study we use a method called the Experience Sampling Method. Students will 
be signaled during the class session and will be asked to fill out a short survey page regarding their 
level of attention to the classroom instruction and the description of their thinking and activities. We 
will also conduct focus group interviews with some of the students participating in the study. During 
the focus group, they will be asked to describe their experiences in the classroom. Parents will be 
asked for an hour of their time for an interview or small group discussion about math and science in 
their child’s school. Neither you nor your child will be paid for participating in this study. The parent 
interviews will be tape recorded (with your permission) by the researcher. 

You and your child will not directly benefit from this study. However, you will indirectly 
benefit by information that is produce from the research that is available to stakeholders at all levels 
(policy makers, administrators, teachers, and parents). There are no anticipated risks involved to being 
a part of this study. Your decision to allow your child to participate in this research study (or to be 
interviewed yourself) is completely voluntary. You are free to participate in this research study or to 
withdraw at any time. You and your child’s privacy and research records will be kept confidential to 
the extent of the law in the David C. Anchin Center (College of Education- USF) to protect your/your 
child’s privacy. We will preserve that privacy by not identifying you, your child, the teacher, or the 
school by name in reports. Authorized research personnel, employees of the Department of Health 
and Human Services, and the USF Institutional Review Board may review the records from this 
research project. 

This study has been approved by the Institutional Review Board of the University of South 
Florida and is monitored by the Division of Research Compliance. Their direct telephone number is 
813-974-5638, should you have any questions. Please review this letter carefully and return it, with 
your signature, to the researcher. If you have any questions about your participation in this research 
study, you may ask the researcher working on this study, or you may contact Dr. Kathryn M. Borman 
at the University of South Florida in Tampa at 813-974-5959. 

By signing the consent form on the back of this letter, you verify that we have informed you of 
the project’s purpose and that you are allowing your child to participate and agree to being 
interviewed. Thank you for assisting us with this research study. 

PLEASE TURN THIS SHEET OVER FOR THE OFFICIAL CONSENT FORM AND SIGNATURE 



Sincerely, 

Kathryn Borman, Ph.D., Principal Investigator 
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Study of the Community Context - Information Sheet for Interviewees 
Assessing the Impact of the National Science Foundation’s Urban Systemic Initiative 

NSF Grant # 9874246 



This research is funded by a grant to the University of South Florida and the David C. Anchin Center 
from the National Science Foundation (Washington, DC). The three-year research grant (1999 - 2001) supports 
an evaluation of the NSF’s Urban Systemic Initiatives in four cities operating USI projects (Chicago, El Paso, 
Memphis, Miami.) The outcomes of the research in the four cities will be compared, both district-wide and in a 
sample of elementary and secondary schools in each site. 

Research Design 

Math and Science Attainment Study (fall and spring 1999 - 200n 

• Analyzes outcome differences in student attainment in science, mathematics, and technology over 
the duration of NSF’s initiatives 

• Determines the extent to which attainment differences between traditionally underserved students 
and their peers have been reduced 

Study of the Enacted Curriculum (spring 2000) 

• Provides analyses of classroom practices and the curriculum in mathematics and science in relation 
to district-adopted standards and systemic reform goals 

• Uses common survey instruments and additional data collection methods across a sample of 
schools and classrooms in each of the four districts. 

Policy Study 0999-2001) 

• Examines the impact of national, state and local policies on local USI reform efforts 

• Examines the extent to which local communities mobilize resources in and outside school to equip 
students to achieve high standards. 

• Three tiered study - 1) policy and resources at the district and local school levels; 2) study of school 
community context; 3) district and school-level impacts 

School Sample 

At each of the four sites, twelve schools (elementary, middle/junior, and high schools) and 
approximately 60 math/science classrooms within those schools were selected. This year’s part of the study 
focuses on two high schools in each site and two teachers (one math and one science) in each of these schools. 
For one week in each of the high schools, students participate in a Study of Student Engagement during their 
class periods and in a focus group at the end of the week. Parental consent forms are necessary for students to 
participate in t his study. There is also a qualitative Study of the Community Context, where partners, 
stakeholders, parents, community residents, and other individuals are contacted and interviewed regarding their 
perceptions of the impact of math and science reform on student achievement and attainment in the district and 
in the individual schools. 

Information for Interviewees 

We will be asking you a series of questions to detemiine the level to which partners and stakeholders have 
been mobilized to support math and science reform at the high school or in the district. We ask your permission 
to tape the interview, which should last no longer than one hour. 

• The first set of questions asks for information about your organization and your responsibilities. 

• The second set of questions asks for your perception of the “community” context - what is the 
community? Do stakeholders and parents/students define community the same way? How is the 
“community” involved in math and science reform? 

• The third part of the interview asks you to think about the individuals, organizations, and other entities 
that might be considered “partners” or stakeholders for the individual school or for the school district. 
We will ask you to brainstorm for partners and then evaluate the effectiveness and importance of each 
of these partnerships. 

• For parents, there are additional questions about perceived changes in instruction, homework, testing, 
etc. since they were students. We also ask parents to identify neighborhood or community level 
resources that exist to support the educational achievement of their children. 
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Purpose 

In order to mask and proteet the identities of the many individuals who eompleted surveys, 
eonsented to interviews, partieipated in foeus groups, and participated in the many other data 
collection activities, a numerical code was assigned to each individual. The information is 
stored in FileMaker Pro and Microsoft Access databases. 

This process created four similar coding systems for the participants in our research: schools 
and teachers, the stakeholders in the Study of the Community Context, principals and district 
administrators, and the teachers and students from the Student Engagement Study. The 
coding logic follows. 



Case Identity Coding System 
Schools and Teachers 
Example: EP2302 

A case identity coding system was developed for schools and teachers using the following 
descriptors: 

City Code 

CH= Chicago 
EP= El Paso 
ME= Memphis 
MI= Miami 



Followed by Numbers 

The first number indicates the level of the school (EP2302) 

I = Elementary 
2= Middle School 
3= High School 

The next number identifies a specific school from one to four, since there were not more than 
four schools at any level. (EP2302) 

The last two numbers identified the teachers, 01, 02, 03, etc. (EP2302) 
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Case Identity Coding System 
Stakeholders in the Study of the Community Context 
Example: CH320121013 

A case identity coding system was developed for individuals interviewed during the Study of 
the Community Context (fall 2000 and spring 2001). This system helps ensure the 
confidentiality of all informants. 

City Code 

EP= El Paso 
CH= Chicago 
ME= Memphis 
MI= Miami 

Followd by Numbers 

The first number indicates the level of the school (CH320121013) 

1= Elementary 
2= Middle School 
3= High School 

The next number identifies a specific school from one to four, since we did not more than four 
schools at any level. (CH3 20 1 21013) 

The third number (zero) is a placeholder. (CH320 121013) 

Tape or File Identification Number 

The next three digits correspond to the file number (in the Access database) for the taped or 
non-taped data. (CH320121013) 

The next number indicates whether the file is from a taped interview or not. 

(CH320121013) 

0 = no tape 

1 = tape 

The following number is the protocol used in the interview. 

(CH320121013) 

1 = School-level 

2 = Stakeholders outside of the school 

3 = Parents, alumni 

4 = No protocol used 

The last number indicates the kind of stakeholder who was interviewed. 

(CH320121013) 

1 = District-level staff 

2 = School-level staff (includes administration, counseling, etc.) 

3 = Teacher 

4 = Parent/guardian 

5 = School governance organization (LSC, SLC, PTA, PTO, EESAC, EClC, etc) 

6 = Volunteer 

7 = Community based organization 

8 = Alumnus 

9 = Multiple categories (i.e. an alumnus who is also a volunteer) 
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Case Identity Coding System 
Principals and District Administrators 
Example: EPI418115 

A case identity coding system was developed for principals and district administrators during 
the 1999-2000 data collection for the Policy Study. 

City Code 

CH= Chicago 
EP= El Paso 
ME= Memphis 
MI= Miami 



Followed by Numbers 

The first number indicates the level of the school. (EP14 18115) 

1= Elementary 
2= Middle School 
3= High School 

The next number identifies a specific school fi-om one to four, since we did not more than four 
schools at any level. (EP14 18115) 

Tape or File Identification Number 

The next three digits correspond to the file number (in the Access database) for the taped or 
non- taped data. (EP 14181 1 5) 

The next number indicates whether the file is from a taped interview or not. 

(EP1418115) 

0 = no tape 

1 = tape 

The last number is the protocol used in the interview. 

(EP1418115) 

4 = Principal interview 

5 = District interview 
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Case Identity Coding System 
Teachers and Students from Student Engagement Study 
Example: CH310206 

A case identity coding system was developed for students of participating teachers that were 
observed during the 1999-2000 data collection for the Student Engagement Study. 

City Code 

CH= Chicago 
EP= El Paso 
ME= Memphis 
MI= Miami 



Followed by Numbers 

The first number indicates the level of the school (CH3 1 0 1 206) 

1= Elementary 
2= Middle School 
3= High School 

The next number identifies a specific school from one to four, since we did not more than four 
schools at any level. (CHS 101206) 

The next two numbers identified the teacher (CHS 101206) with the third digit indicating the 
period of the classroom observation, from first through seventh. (CHS 101206) 

The final two digits (CHS101206) identify the students who participated in the Student 
Engagement Study. Values ranged from 01 through 16. 




17 



19 




20 



BEST COPY AVAILABLE 



Purpose 

A three-part observation protocol was used to collect data on instructional practices occurring 
in the classrooms of participating teachers. The first part included information about the class 
such as the number of students, the ethnicity and gender of the students and the teacher. The 
second part identified the instmctional materials, including technology, used during the 
observed lesson, and how the class was arranged. Finally, we noted classroom events, such as 
teacher and/or student conversations, interactions and activities. 



For Wliat Study? 

The classroom observations are a part of the Mathematics/Science Attainment Study. During 
fall 1999 and spring 2000, 188 classroom observations were conducted in 47 schools fi'om the 
four cities participating in the research study. 



Relationsliip to Other Parts of the Study 

As a part of the Mathematics/Science Attainment Study, the classroom observations provided 
evidence of the implementation of Driver 1, standards- based instmction. 



Development Process 

In the fall of 1999, the research team designed the classroom observation protocol to collect 
information related to Driver 1, standards-based instruction. The process began by 
identifying the categories of standards-based practices for observation, using standards 
developed by the National Council of Teachers of Mathematics (NCTM) and the National 
Academy of Science as guides. Based upon that identification, the research team organized 
the protocol into the three parts: information about the class, information about instmctional 
materials, and observation of classroom events. 

Concepts Infonning the Instmment 

The standards developed by the National Council of Teachers of Mathematics (NCTM) and 
the National Academy of Science emphasize high-quality mathematics and science education 
“for all students.” They are also tied to the notion of the importance of resources to the 
enhancement of classroom instmction. Students cannot be taught with equity and fairness 
unless schools and classrooms are equipped with the necessary technological support, in 
addition to instmction that is standards-based and carried out by teachers who have 
appropriate certification and professional development experience. 



Observation Protocol 



Part 1: Classroom Data: 

While in the classroom, please gather the foUowing bits of information to place the 
observations in the context of the classroom: 

1) What are the lesson plans for the day? 

a) Where does this fit in with what the class has been doing (the past week)? 

b) Where does this fit in with what the class will be doing (the next week)? 

2) Age range of the students 

3) Gender of the students (# of females/males) 

4) Ethnicity of the students (how many of each ethnic group) 

5) Indicators of socio-economic status of the students 

6) Approximate age of the teacher 

7) Gender of the teacher 

8) Ethnicity of the teacher 



Part 2: Physical Description of the Room and its People: 

1 ) Seating arrangement - make a map 

2) Position of teacher in classroom (take note on how this changes throughout the class 
period) 

3) Decoration of classroom , 

4) Textbooks used 

5) Technology used 

6) Other materials 



Part 3: Description of Classroom Events: 

In your fieldnotes indicate the type of instmctional activity that is occurring: 

• T = Teacher centered activity (lecture or teacher centered explanation) 

• TS = Teacher - Student activity (questions, teacher moderated dialogue) 

• S = Student centered activity (student presentation, group project, joint work) 
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Purpose 

One of the instruments utilized to code the classroom observations was a classroom 
observation checklist. The checklist consisted of mathematics and science standards as well 
as components of other instmments designed to identify five mathematics and science 
standards-based practices (Sykes, 1999; NCTM, 1991; NRC, 1996). This instrument 
identifies which standards the teachers were using as part of their instmctional activities in the 
classrooms. It was used by field research staff to code observations from fall 1999 and spring 
2000 classroom data collection. 

For W'liat Study? 

The classroom observation checklist was used to analyze the 1 88 classroom observations in 
the Mathematics and Science Attainment Study in fall 1999 and spring 2000. 

Relationship to Other Paits of the Study 

One of the objectives of the larger evaluation study, Assessing the Impact of the National 
Science Foundation ’s Urban Systemic Initiative, is to provide evidence of the implementation 
of Driver 1 (Standards-based Instmction). While the classroom observations documented all 
aspects of classroom instmction, the classroom checklist focused the observations on the five 
standards; communication, problem solving, representation, reasoning, and connections. The 
results of the checklist will provide evidence of the level of standards-based instmction 
occurring in the classrooms of the participating teachers. 



Development Process 

The classroom observation coding checklist and classroom observation protocol were created 
at the same time in the fall of 1999. Once the five mathematics and science standards were 
identified, they were organized to include three parallel statements describing teacher actions 
on a 0 to 3 scale: the first referred to whether the teacher modeled the particular standard; the 
second to whether the teacher provided students with the opportunity to engage in the process; 
and the third to whether the teacher provided appropriate guidance to students. 

Concepts Infonning the Instrument 

Concepts informing this study are derived from the NCTM standards: 

• Communication is a way of sharing ideas and clarifying understanding. 

• Problem solving is engaging in a task for which the solution method is not known in 
advance. (Problem solving and inquiry were used interchangeably in this study.) 

• Representation applies to processes and products that are observable externally as well 
as to those that occur "internally," in the minds of people, such as diagrams, graphical 
displays, and symbolic expressions. 

• Reasoning involves developing ideas, exploring phenomena, justifying results, and 
using mathematical or scientific conjectures in all content areas. 

• Connections are the interplay among mathematical or scientific topics, in contexts that 
relate those topics to other subject areas, and in students’ own interests and 
experiences. (NCTM, 2000) 
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1 . Observer’s Name 


71 

Classroom Observation Checklist 

8. Teacher Gender 1 


2. Date and Time 


9. Teacher Ethnicitv 


3. School Name 


1 0. Teacher Age (approx) 1 


4. Teacher Name 


11. Class Size i 


5. Grade Level 


12. Ethnicitv of Students (#'s) i 


6. Subject 


13. Gender of Students (#'s) 1 


7. Class Academic level (if applicable) 


Please indicate the emphasis placed on each of the statements: 0=None, l=Minimal, 2=Moderate, 3=Major 


Communication i 


0 12 3 


Modeling appropriate use of mathematical and/or scientific language to communicate | 
mathematical and/or scientific concepts. (Look for effective communication j 
techniques, oral, written and visual forms). j 


0 12 3 


Providing students with the opportunity to communicate mathematical and/or I 
scientific ideas using oral, written and/or visual forms, (e.g. students explaining the 1 
process by which they reached a solution to a mathematical or scientific problem.). 


0 12 3 


Guiding student communication of mathematical and/or scientific ideas, (e.g. telling a | 
student “That wasn’t completely clear, could you rephrase that?”) I 


Problem Solving i 


0 12 3 


Modeling the use of problem solving strategies above and beyond presenting a I 
specific technique or procedure, (e.g. presenting a variety of strategies which can be | 
used to solve a problem and encouraging students to experiment). 1 


0 12 3 


Providing students with the opportunity to work alone or in groups to discover | 
solutions to problems. | 


0 12 3 


Guiding students in exploring ways to solve a problem and/or in discovering the | 
answers to questions (e.g. teacher asks questions to stimulate student thinking; asks j 
students to share the different approaches they may have used.). 1 


Reoresentation 1 


0 12 3 


Modeling the use of mathematical and scientific representations, (e.g. using visual 1 
and/or tactile representations (graphs, tables, models) of mathematical and/or I 
scientific concepts to help explain these concepts to students). 1 


0 12 3 


Providing students with the opportunity to use mathematical and/or scientific I 
representations individually and/or in groups to solve problems. 1 


0 12 3 


Guiding students in their use of mathematical and/or scientific representations to I 
solve problems. I 


Reasoning i 


0 12 3 


Modeling orally the mathematical and/or scientific reasoning processes (e.g. I 

explaining the logic behind each step in a procedure or formula; explaining the reason 1 
why situations occur utilizing knowledge the students have been taught.). I 


0 12 3 


Providing students with the opportunity to develop reasoning skills, (e.g. asking I 
students to explain the procedure they used to solve a problem, asking students “why” I 
questions for which the student has not been taught the answer). I 


0 12 3 


Guiding students use of reasoning, (e.g. when students explain the procedures they 1 
used to solve a problem, teachers question further: “Whv does in this 1 

situation”). I 


Connections 1 


0 12 3 


Presenting and explaining one or more of the following: 1 

1. the ways mathematical and science concepts covered in class can be and j 

are used to complete real world tasks | 

2. the connections between various topics within mathematics and/or science 1 
and/or between mathematics and/or science and other disciplines 

3. the connections between the current lesson and mathematical and/or 

scientific concepts previously covered j 

22 1 
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0 12 3 




0 12 3 



0 12 3 



Providing students with the opportunity to explore, present and/or discuss one or 
more of the following: 

1. the ways mathematical and science concepts covered in class can be and 
are used to complete real world tasks 

2. the connections between various topics within mathematics and/or science 
and/or between mathematics and/or science and other disciplines 

3. the connections between the current lesson and mathematical and/or 
scientific concepts previously covered 

Guiding student exploration and discussion of one or more of the following: 

1. the ways mathematical and science concepts covered in class can be and 
are used to complete real world tasks 

2. the connections between various topics within mathematics and/or science 
and/or between mathematics and/or science and other disciplines 

3. the connections between the current lesson and mathematical and/or 
scientific concepts previously covered 

Classroom displays depict the ways mathematical and/or science concepts covered in 
class are applied to real world tasks. 






23 



25 



PuiT>ose 

The Authentic Instructional Practices Coding Matrix was used to code the classroom observations in 
NUD*IST4.0. 



For What Study? 

The coding matrix was used for analysis of the classroom observations in the Mathematics and 
Science Attainment Study during fall 1999 and spring 2000. 

Relationship to Other Parts of the Study 

The coding matrix is one of the instruments used for analysis of Driver 1, standards -based instruction. 
The findings from the analysis of the classroom observations using the coding matrix will identify 
whether teaching practices are teacher centered (didactic), subject centered (conceptual), or student 
centered (constructivist). Student centered (constructivist) is one indicator variable identified for 
Driver 1 in the study’s structural equation model. 



Relationship of Items to NSF Drivers 



Instrument 


Sample 


Driver 1 


Driver 

2 


Driver 

3 


Driver 

4 


Driver 

5 


Driver 

6 


Proposed 
Driver 7 


Authentic 

Instructional 

Practices 

Coding 

Matrix 


Teachers 


All 

Questions 















Development Process 

The Authentic Instructional Practices Coding Matrix was developed by the research team in the spring 
of 2000, as an instrument to analyze classroom observations. The coding matrix was developed using 
two major sources: Authentic Instruction Classroom Observation Form (Borman, G, Rachuba, L, 
Datnow, A, Alberg, M, Maciver, M, Stringfield, S, & Ross, S:, 2000; D’Agrostino, 1996, the 
Secondary Teacher Analysis Matrix (Simmons, P.E., Emory, A., Carter, T., Coker, T, 1999). Using 
these sources as a guide, the Authentic Instructional Practices Coding Matrix is composed of four 
categories: classroom communication, social support, student engagement, and lesson coherence. 

Concepts Infonniim the Instrument 

Unlike traditional approaches to teaching, authentic instruction involves a focus that is primarily 
student-centered and emphasizes constructivist approaches to teaching and learning. In a student- 
centered or authentic instructional scenario, students negotiate with their teacher an understanding of 
the important ideas in the lesson based on student input and discussion to develop a deep, 
comprehensive understanding of concepts. In such instructional milieus, we would expect to see the 
teacher facilitating student-centered communication , social support, and student engagement with the 
instructional material at hand. In addition, the lesson itself would evince activities, concepts and ideas 
that are significant to the instructional topic and that are covered in a manner that enhances student 
understanding. 
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Authentic Instructional Practices Coding Matrix 


Types of 

Classroom 

Processes 


1. Teacher Centered 
(Didactic) 


2. Subject Centered 
(Conceptual) 


3, Student Centered 
(Constructivist) 


Communication 


The teacher is "telling." 

Content tends to be 
descriptive with little 
emphasis on explanation. 
There is no probing of student 
responses and no discussion 
of the lesson material. The 
teacher may ask questions 
that ask students to recall 
facts. There is very little 
communication between 
students or from students to 
teacher. 


The teacher explains and 
analyzes the content 
emphasizing procedural 
knowledge with explanations, 
and conceptual understanding 
maybe included. The teacher 
probes for student responses, but 
there is little conversation 
occurring among students — 
most dialogue is directed 
through the teacher. 


Students are encouraged to converse 
among themselves about the lesson 
material. The teacher may or may not 
engage in the conversations. The teacher 
and students negotiate understanding of 
key ideas based on students' ideas and 
content. Investigations dominate content. 
There is evidence that the purpose of 
sharing ideas is to arrive at a deeper 
understanding. Conceptual content and 
connections are embedded into the 
design, implementation, analysis, and 
report of investigations. 




ZERO 


ONE 


TWO 


THREE 


Social Support 


The rapport between 
teacher and students 
is not good. The 
working relationship 
between teacher and 
students is rarely 
constructive. The 
overall atmosphere of 
the classroom is 
negative. 


Support is mixed. Teacher 
praises students 
occasionally. At other 
times student effort goes 
unnoticed. Students are 
not encouraged to support 
one another. 


Support is usually positive. 
Teacher-student rapport is 
good. There is some 
evidence of high 
expectations for learning 
and trying hard. Teacher 
focuses on student 
successes and does not 
dwell on failures. 


A strong friendship and 
mutual trust exists 
between teacher and 
students. The 
atmosphere clearly 
supports student effort. 
Lowest achieving 
students receive support 
from all. 


Student 

Engagement 


Students appear to be 
inattentive. They 
may look as though 
they are bored or 
preoccupied with 
thoughts unrelated to 
the task at hand. One 
or a few students 
may be disruptive. 


Students appear to be 
occasionally on-task. For 
those that are on-task, 
however, they seem to be 
rather lethargic and/or not 
trying very hard. 


Students for most of the 
time are on-task pursuing 
the substance of the lesson. 
Students have, however, 
occasional lapses in 
concentration. Students, 
with few exceptions are on 
task. 


All but one or two 
student(s) are deeply 
engaged in the lesson 
(paying attention, 
clearly interested in 
learning the material, 
concentrating) for all 
but a few short instances 
of the lesson. 


Coherence of 
Lesson 


Material is presented 
in superficial 
fragments with very 
little connection 
between parts. 


Some activities focus on 
significant topics, but other 
key concepts or ideas are 
not appropriately covered. 
The lesson activities are 
not well connected to over- 
arching concepts or ideas. 
(Activities for the sake of 
doing activities.) 


Some key concepts or 
ideas are covered in depth. 
There are activities that 
focus on significant topics 
that are key to the whole 
lesson content, but 
coverage is uneven and 
other key concepts or ideas 
are not appropriately 
covered. 

There are some good parts 
of the lesson, but there are 
parts that are missing or do 
not appropriately support 
concept development. 


Key concepts/ideas are 
covered in depth. The 
lesson content is 
presented as a whole, 
and is structured in a 
way that allows for the 
sequencing and 
structuring of a complex 
topic. Each topic 
appears to build on 
another in an effort to 
foster deeper student 
understanding. 
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Purpose 

In this study, three instruments were used to determine what students are doing, thinking, and 
feeling throughout the eourse of a elass: the student engagement survey, the student foeus 
group, and the parent survey. The student engagement survey was used to examine how 
classroom activities are related to student engagement. The focus group concentrated on 
learning about the relationship between mathematics and science in their everyday life, and 
future plans. The parent survey sought to elicit feedback on how their child’s mathematics 
and science courses influence their home life, and after school activities. In addition, . 
classroom observations were used to contextualize the student engagement survey. The 
classroom observation protocol used in this study is the same used for the classroom 
observations conducted in fall 1999 and spring 2000 (see page 3). The Student Engagement 
Study began in the fall of 1999 in two high schools from Chicago and Memphis with three 
participating teachers from each school. In spring 2000, the research continued in two high 
schools from Miami and El Paso with two participating teachers. Three hundred and four 
students in a total of eight schools participated in the Student Engagement Study. 

For Wliat Study 

These instruments were created to collect student information for the Student Engagement 
Study, nestled in the Mathematics/Science Attainment Study. This study assesses the impact 
of classroom activities on high school engagement in on-going mathematics and science 
teaching and learning activities using a methodological approach called Experience Sampling 
Method (ESM) developed by Csikszentmihalyi and Larson. This study aims to provide an 
understanding of student achievement and engagement beyond students’ performance on high 
stakes, standardized state tests. 

Relationship to Other Parts of the Study 

This study adds a level of information that is not captured by other studies conducted in our 
project. Students were asked to complete the engagement survey when they were signaled 
several times during classes; thus, the instrument provides time- series data on student 
engagement moment- to- moment. Using Experience Sampling Method (ESM), this study 
facihtates an analysis of the level that is immediately important for student learning: 
individual engagement. This study, through the use of the instruments mentioned above, 
provides student- level information on Driver 1, Standards- based Instmction and how 
instmction relates to student engagement. Driver 4, Mobilization of Stakeholders, is 
discussed during the focus group and responded to by parents in their survey. Drivers 5 and 
6, Increased Attainment and Reduced Differences in Achievement, were addressed by the 
students in the focus group and on the Survey of the Enacted Curriculum. 
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Relationship of ESIVI Items to NS 



Drivers 



Instrument 


Sample 


Driver 1 


Driver 2 


Driver 3 


Driver 4 


Driver 5 


Driver 6 


Proposed 
Driver 7 


ESM Student 
Survey 


Students 


1,3,4 








2,5 


2 




ESM Focus 
Group 
Questions 


Students 








4 


1,3 


1,3,4 




Parent 

Survey 


Parents 








1,5, 6, 7, 
8, 11, 12 


2, 3, 4, 
11, 12 







Development Process 

All three instruments were created in fall 2000 by the research team. The Experience 
Sampling Method (ESM) was modified to focus on student engagement in mathematics and 
science classroom activities during their regularly scheduled classes. Recent research by 
Shemoff, et al. (2000) and Yair (2000) used ESM to study youths’ life experiences in general, 
not specifically for studying classroom experiences. Our research was designed based upon 
the insights and methods derived from these previous studies, but adds methodological 
approaches for studying classroom processes. Since we wanted to gain insight into students’ 
responses to classroom events over the duration of several classroom lessons, we observed the 
students, and created the survey questions to enable the students to record their impressions in 
real time. Questions for the focus groups were designed to elicit narrative feedback from 
students who had participated in the Student Engagement Study throughout the week. A 
survey for parents of students in the study was also created in an attempt to elicit additional 
parental feedback on the impact of mathematics and science reform at the individual, family, 
or school level. 

Concepts InfoimiiiR the Instalment 

We studied the ESM instrument developed by Csikszentmihalyi, Schneider, and their 
colleagues at the University of Chicago as a model for the instruments in the Student 
Engagement Study. Their instrument was designed to capture students’ life both inside and 
outside classrooms and schools. We modified it, so we could focus on what is important in 
classroom situations. 
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Student Engagement Study 

Student Survey Form (fllled out each time the student was signaled) 



What is your beeper ID? ( ) 

When you were signaled the first time today, 
What was the main thing that you were doing? 

What else were you doing? 

What was on your mind? 



Please fill in the bubbles (example O) Strongly Disagree = SD, Disagree = D, Agree=A, Strongly Agree=SA 
When you were signaled the first time today, 

SD D A SA 

I was paying attention to class O O O O 

I did not feel like listening O O O O 

My motivation level was high O O O O 

1 was bored O O O O 

I was enjoying class O O O O 

I was focused more on class than anything else O O O O 

I wished the class would end soon O O O O 

I was completely into class O O O O 

Please fill in the bubbles. 

When you were signaled the third time today, was the main thing you were doing more like 
Work O Play O Both O Neither O 

What was being taught was Very easy O Easy O About right O Difficult O Very difficult O Can ’t tell O 

What was being taught was something that I already knew YesO No O Can’t tell O 

Please fill in the bubbles that apply. 

When you were signaled the third time today, what was being taught was important for 
my everyday life YesO No O 

going to college YesO No O 

my future job YesO No O 

future tests YesO No O 

If you felt that class at the time of signal was important for tests, please tell us for which tests it was important. Fill in all that 
apply . 

Class quiz O Midterm/Final O SAT or ACT O 
State Assessment Tests (e.g., TAAS, End of Course Exam) O 

Class was unrelated to test O 



Please fill in the bubbles that apply. 



When you were signaled the third time today, you were feeling: (Fill in all that apply ) 
HappyO ConfusedO ActiveO Having funO 

NervousO IntimidatedO SadO CooperativeO 

RelaxedO WorriedO AngryO ConfidentO 

CompetitiveO FrustratedO BusyO SleepyO 

Were you talking with anyone? 

No oneO ClassmateO Teacher O 
Was it about the class? YesO No O 



Comments - if any: 
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USI STUDENT ENGAGEMENT STUDY 
QUESTIONS FOR FOCUS GROUP INTERVIEWS 



F ORMAT : 8-10 students in a group 

2 moderators - one to record and one to facilitate 
Tape recorders around table to capture narrative 

The final questions will derive from the feedback and observations of the week’s Student 
Engagement Study. We will ask the schools if we can have 1 - 1 !4iours with the students. 
We need at least one hour (or one class period) in order to conduct the group properly and 
give all students a chance to speak. 

1 . Coursework - What math and/or science classes are you taking this year? (Note: 
This question will not take time if we focus on math and science. The information 
is necessary to help answer our research questions related to Drivers 5 and 6.) 

2. Are you and your friends in the same math and/or science classes? 



3. What kinds of extracurricular activities (clubs, sports, etc.) are you involved in at 
school? (Drivers 5 & 6) 



4. What kinds of activities, organizations, etc. are you involved with in your 
neighborhood, in the community? What do you do with these groups? Do you 
think your work with them helps you in school? How? (Drivers 4 & 6) 

5. What do you intend to do after high school? 

a. Go to vocational school? 

b. Go to a community college? 

c. Go to university? 

d. Go into the workforce 

e. Other 



6. Do your future plans involve using math, science, or technology? 
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Parent Survey- Studeot Engagement Study 



What is your relationship to the high school student who participated in the Student Engagement Study? (Please check one 
below). 



Mother 


Father 


Guardian 


Grandparent 


Other relative 


Other relationship 



How far did you go in school? Please check one below. 



Finished elementary school 

Finished some high school 
Finished high school 



Some Vocational/technical 

education after high school 

Some community college, 

college, or university courses 



_____ Completed a bachelor’s degree 
at a college of university 

Completed a professional or 

advanced degree 



Please tell us the number of children in your household. Please check below. 
Ages 0-4 years 



_Ages 5-10 years 



Ages 11-13 years 

Ages 14-18 years 



Have you ever encouraged your teenager to get a book^ a manual or a computer progranty or to take a course that would 
help him/her to prepare for any of the following tests? 



State Assessment Tests (FCAT, TAAS, TCAP, Armed Services Vocational Aptitude Battery (ASVAB) 

CASE, IGAP) 

American College Test (ACT) General Education Development Test (GED) 

Scholastic Aptitude Test (SAT) 



Do you see your child using his/her math and/or science knowledge around the house?YQS No 

Do you see your child using his/her math and/or science knowledge at his or her joblY^s No 1 

How many times in the past year have you talked to your teenager about going to worky entering the militaryy or applying to a 
vocational/technical school, college, or university for education after high school? 

Never 1 or 2 times 3-6 times at least once a month 

Mark any of the following ways you have helped your teenager make decisions about what path to follow after high school? 

Offered assistance, but our teenager wants to do this himself/herself 

Talked to our teenager about particular schools 

Gave our teenager information (brochure, booklet) that we received from school 

Offered to take our teenager to a private college counselor 

Referred our teenager to the guidance counselor at school 

Referred our teenager to educational outreach programs like Talent Search, Upward Bound, ora Bridge program 

Please put a check by the resources that are available in your neighborhood to help your child with school work? 

Tutorial programs After-school youth programs Other 

Library Community Center 



Please check any of these activities or events you have participated in during the past year. 

Math fairs or science fairs 

Workshops for parents on homework assistance PTA/PTO, LSC meeting 

Workshops on understanding changes in math Conference with teacher 

and science A classroom visit 

Family math or family science nights at the A school event (play, concert, sports competition, 

school honor ceremony, etc.) 
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Please check all the ways that you get information from your child’s school about his or her math or science cla 



Flyers 

Letters 

Phone calls 

School newsletter 



School newspaper 

My child tells me 

Other: 



Please check any changes you have seen in math and/or science courses since your child started high school? 

More math classes to take to graduate 

More science classes to take to graduate 

Algebra in the freshman year 

Harder (or more challenging) classes 

Easier classes 

Other: _ 



Is homework different from the homework that you did in high school? If so, please check the ways it is 
different. 

More homework 

Harder homework 

Less homework 

Can bring books home 

Cannot bring books home 

More worksheets 

How important to you is it for your child to take the following courses? Mark each one from 0=not important to 
3=very important 

Algebra I 
Algebra II 
Geometry 
Pre-calculus 
Biology 
Chemistry 
Physics 
Earth Science 
Calculus 



What classes are your child is taking this semester? (Please mark all that apply and add ones that we left 
off). 

I don’t know 

English 

A Igebra I 

Algebra II 

Geometry 

Pre-calculus 

Calculus 

Biology 

Earth science 

General science 

Integrated science 

Marine science 

Environmental science 

Physical science 

Anatomy and physiology 

Chemistry 

Physics 

American History 

_ World History 

Civics/Government 

Art/music 

Foreign language 

ROTC 

Shop classes - which one? 

Computer classes 

Other: 
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Puroose 

The purposes of this survey were to determine whether the quality of professional 
community as measured by the School Culture Quality Survey (SCQS) (Katzenmeyer, 
1 999) was positively related to student achievement in mathematics, and whether a 
school culture driver added to the NSF driver model might enhance our ability to predict 
the values of the outcome drivers. 



For What Study 

The survey, as part of the Mathematics and Science Attainment Study, sought to 
determine whether or not significant relationships exist between measures of teachers’ 
professional community (school culture) and student’s achievement in Mathematics. 
Mathematics achievement scores, school demographic data, and the quality of 
professional community scores from 39 schools located in Chicago, El Paso, Memphis, 
and Miami were studied. 

Relationship to Other Parts of the Study 

The proposed addition of Driver 7, School Culture, became apparent after initial analysis 
of the data. The survey, based on data from 39 schools in Chicago, El Paso, Memphis, 
and Miami, demonstrates that the quality of the Professional Community of the school, as 
measured by the SCQS is strongly related to school level mathematics gain scores 
adjusted for pretest scores, differences in percentage of students on free/reduced lunch, 
and school mobility rate (Drivers 5 and 6). 



Relationship of SCQS to NSF’s Six Drivers 



Instrument 


Sample 


Driver 

1 


Driver 

2 


Driver 

3 


Driver 

4 


Driver 

5 


Driver 

6 


Proposed Driver 7 


School 

Culture 

Quality 

Survey 


All staff - 

school 

level 






14, 

24, 

31,32 








1,2, 3,4, 5,6, 7, 8, 9, 
10, 1 1, 12, 15, 16, 17, 
18, 19, 20,21,22, 23, 
2,27, 28,29, 30, 33, 
34, 35, 36 



Development Process 

Based on ideas and concepts from the quality movement and the work of Peter Senge, the 
SCQS conceives the quality of professional community to include the following factors: 
shared vision, facilitative leadership, teamwork, and learning community. Beginning in 
1 996, the SCQS was developed to assess the extent to which a school professional 
community characterized by these four elements is present in schools. Items that 
addressed these four elements were developed. The pencil and paper scan sheet 
approach to data collection was chosen because of a conviction that, to gain acceptance, 
the measure needed to be effective in gathering the needed information, and efficient in 
terms of dollar costs and staff time. 

Information about the complete School Culture Quality Survey is available at 

http://anchin.coedu.usf edu 
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Puiuose 

The study of the enacted curriculum consists of three surveys for teachers and students in 
mathematics with three parallel surveys for science. Mathematics and science teachers 
complete two surveys: survey of instmctional content, and survey of classroom practices. 
Students complete the student survey of classroom practices for either science or 
mathematics. Together, these surveys elicit teacher and student responses on what is 
occurring in mathematics and science classrooms in terms of content, instmction, 
professional development, technology, and homework. 

For What Study 

These surveys are a part of the Study of the Enacted Curriculum, one of the three studies 
that compose the larger research study. Assessing the Impact of the National Science 
Foundation 's Urban Systemic Initiative. The surveys were administered to the teachers 
and students in the four participating cities in spring 2000. One hundred forty- one 
teachers completed the survey, with 4,615 students of the participating teachers 
completing a parallel survey of classroom practices. 

Relationship to Other Paits of the Study 

Information from these surveys will not only inform the Study of the Enacted 
Curriculum, but the Mathematics and Science Attainment Study as well. These self- 
reported data can be compared with the classroom observation data, and lend more 
information to Driver 1, Standards-based Instmction. Additionally, the survey provides 
information on professional development and other resources (Driver 3, Unified 
Application of Resources). The surveys can also determine if there are differences in 
mathematics and science teaching that are related to state policy initiatives and state 
standards (Driver 2, Unified Set of Policies). 

Development Process 

The Study of the Enacted Curriculum was a subcontracted study through the Council of 
Chief State School Officers. The instmments had been in use since 1999, and 
collaboration with the USE study presented an opportunity to extend the research on 
classroom curriculum and practices by including the schools in our sample. 

Information about the complete Study of the Enacted Curriculum is available at 
http://www.ccsso.org/proiects/proiects.html 
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PuiT>ose 

In spring 2000, researchers carried out interviews with the principals of the 47 schools in 
the four cities participating in the study. The principal interview protocol consists of 1 3 
multi-part items related to NSF’s Six-Driver model. In addition to providing responses to 
questions asked during the interview, our participants were also asked to rate the level of 
importance in implementing reform of specific strategies such as “Identifying teachers 
classroom instmctional needs and providing them support,” or “Evaluating the potential 
value of new instmctional strategies and programs.” 



For What Study? 

The principal interviews were part of the project’s Policy Study and lent information to 
the analysis of the implementation of NSF’s Drivers from an administrative perspective. 



Relationship to Other Paits of the Study 

The principal interview questions were devised to provide data for each of the Drivers in 
the NSF model. The interview protocol elicited information about individuals’ role as 
principal, their definition of standards- based instmction, their views of policies impacting 
mathematics and science, the impact of professional development, social equity issues 
impacting their schools, and the kinds of resources available as well as the coordinated 
use of these resources. All together, the principal interview data can be used to support 
analysis for any of the drivers. 



Relationsliip of Protocol Items to NSF's Six Drivers 



Instrument 


Sample 


Driver 

1 


Driver 

2 


Driver 

3 


Driver 

4 


Driver 

5 


Driver 

6 


Proposed 
Driver 7 


Principal 

Interview 

Protocol 


School level 
administrator 


2 


4,6, 


7, 8, 9, 
10, 12, 


11 


13, 


5, 13 


1 



Development Process 

Items for the Principal Interview protocol were identified and developed by the 1999- 
2000 research team. Questions were developed to elicit feedback from principals related 
to each of the NSF drivers. It was also determined that the administrators would be asked 
to rate the importance of some of the topics on a zero to three scale, with zero being not 
at all important and three as extremely important. This rating scale has enabled the 
principal responses to be quantified for use in the study’s Six-Driver model. 
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Principal Interview Protocol 



General Context 

1. State the main purpose of our study, and the goals of this interview. 

2. Gather information about the principal: 

Name: Age: Gender: _ 



. Race/Ethnicity:_ 



How long have you been a principal? 

How long have you been a principal at this 
school? 

Have you held other administrative positions? 
How many years have you taught? 



What was the area? 

What is your previous non-academic work 
experience? 

What is your highest level of education 



Specific Context 

I would like to ask you about a dozen questions. In addition, I will ask you to use a rating scale on some: 

0 = Not at all important to 3 = Extremely important. 

Q1 : (Driver 7) How would you define your role as principal? Probe: How do you support 

instruction and learning in your school? 

How would you rate the importance of: 

la) Identifying teacher needs and providing them support with regard to instruction 
0 = Not at all important to 3 = Extremely important. . 

Please provide examples of your efforts in this area 

How would you rate the importance of: 

lb) Making program adjustments in response to unmet student needs? 

0 = Not at all important to 3 = Extremely important. 

• Please provide examples of your efforts in this area. 

Q2: (Driver 1) How would you define standards -based instruction? 

2a) How is standards-based instruction implemented in the math and science classrooms at your 
school? 

• Please rate the impact of standards-based instruction on the curriculum 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

Q3: (Multiple Drivers) Please detail changes that have occurred in your school since the 
implementation of the USl. 

• Please tell us the specific ways that the USI has affected the following in both math and 
science: 

3a) Student Achievement 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

• Please provide examples 

3b) Professional Development 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

• Please provide examples 

3c) Technology 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

• Please provide examples 

3d) Social Organization (block class structuring, teacher prep time) 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

• Please provide examples 
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Q4: (Driver 2) Can you identify those policies (national, state, and district) regarding math 

and/or science instruction/achievement that have the biggest impact on your school? 

4a) What do you consider to be your role in implementing national and state policies at your 
school? 

4b) What are the policies that support or hinder the implementation of the USI in your school? 

Q5: (Driver 6) A major policy emphasis in school reform is the reduction of social inequities. 

What social equity issues or problems are particularly troublesome at your school? 

5a) Do you feel that the USI has been useful in addressing such issues/problems? Why or why 
not? 

Q6: (Driver 2) A lot of emphasis is placed on accountability and evaluation in reform. As a 

principal, what measures or processes do you use to assess teacher performance, either 
formally or informally? 

• Please rate its impact on the classroom. 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 “ Not at all important to 3 = Extremely important. 

Q7: (Driver 3) During the last three years have you or your staff attended any professional 

development activities? 

7a) When during the year? What topics or programs were covered? Who sponsored the 
professional development? 

7b) What are the percentages of staff who attended? For staff who do not attend, how is 
information disseminated? 

7c) How many of your teachers are earning advanced degrees? Do you offer tuition 
reimbursement? 

Q8: (Driver 3) How many out-of-field teachers are there in your school? What is the impact of 

the out-of-field teachers on meeting the USI goals? 

Q9: (Driver 3) How often are meetings among staff at your school held to discuss instructional 

practices, and other issues related to the USI? (Frequency, % of attendees, topics). 

• Please rate the impact of these meetings on the math and science programs at your school 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

QIO: (Driver 3) Has participation in the USI enabled you to increase the amount of available 

resources (technology, instructional materials, funding, time). Provide examples. 

Qll: (Driver 4) Thinking about community resources such as university partnerships, businesses, 

parents, etc., how has their participation in USI-related or other reform efforts impacted 
your school? (What are the other reform efforts?) 

• Please rate their participation as it impacts the science and/or math programs in your school. 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

Q 12: (Driver 3) How are resources coordinated at your school and within the school district? 

• Please rate the impact of this coordination on the science and/or math programs in your school. 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 

Q13: (Driver 5/6) How do you know if students are benefiting from your schools involvement in 

the USI? 

13a) Are there particular achievement markers of student achievement/accomplishment you 
consider important? 

• Please rate the impact these have on your school. 

MATH 0 = Not at all important to 3 = Extremely important. 

SCIENCE 0 = Not at all important to 3 = Extremely important. 
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Purpose 

From January to May 2000, researchers conducted interviews with district level staff and 
administrators in the four sites (and six districts) participating in the study. The district 
adrninistrator interview protocol consists of 14 multi-part items related to NSF’s Six- 
Driver model. In addition to providing responses to questions asked during the interview, 
participants were also asked to rate the level of importance in implementing reform of 
specific strategies such as “Identifying teachers classroom instmctional needs and 
providing them support,” or “Evaluating the potential value of new instmctional 
strategies and programs. 



For What Study? 

The district administrator interviews were part of the project’s Policy Study and lent 
information to the analysis of the implementation of NSF’s Drivers. 



Relationship to Other Parts of the Study 

In addition to providing data to inform the complete policy study, responses from the 
interviews with district administrators provided alternative input for analysis of teacher 
professional development and interviews with principals. 



Relationship of Protocol Items to NSF’s Six Drivers 



Instrument 


Sample 


Driver 1 


Driver 2 


Driver 3 


Driver 4 


Driver 5 


Driver 6 


Proposed 
Driver 7 


USI District 
Interview 
Protocol 


District and 
school level 
administrators 


1 


la, 2b, 
2c, 4, 4a, 
4b, 4c, 
4d, 4e, 
8a 


le, 2f, 
2g, 3a, 
3b, 6, 

6a, 6b, 
6c, 7, 7a, 
7b, 7c, 
7d, 7e, 
7f, 8b, 9, 
10, 10b, 
10c 


Ic, Id, 
le, 12, 
12a 


13, 13a 


13 


6d, 7g 



Development Process 

Items for the district administrator protocol were identified and developed by the 1999- 
2000 research team. Questions were developed to elicit feedback from district- level staff 
related to each of the NSF drivers. To distinguish between curriculum specialists 
(“Curr”) and other district-level administrators (“All”), a column labeled “Role” was 
added to the left of the questions. This distinction was made due to the content of a 
couple questions, namely, questions two and nine. These questions required knowledge 
that curriculum specialists would possess, and not necessarily other district 
administrators. 
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USI District Interview Protocol 

General Context 

State the main purpose of our study, and the goals of this interview. 

Review the Informed Consent form. 

Gather information about the district person: 

Age: Gender: Race/Ethnicity:_^ 



2 . 

3. 

Name: 



o 

ERIC 



How long have you been working at the district level? How long have you been in this 
position? 

What other administrative positions have your held? 

How many years did you teach, and what was the subject area? 

What previous non-academic work experience have you had? 

What is your highest level of education? 



40 



Role 


0# 


Questions 


Driver H 


All 


1 


Based upon your experiences during the past several years, what do you believe is the 
best way to improve mathematics and science education in your district? 


1 1 




la) 


Currently, national efforts to improve education use the term reform. What do you believe 
these national reform initiatives are and how do they apply to your district? 






lb) 


The National Science Foundation, in the Urban Systemic Initiative, defined reform as 
standard s-based reform that includes implicit constructivist principles. To what extent does i 
the USI definition of reform support or conflict with your views? 






ic) 


(Driver 2, 3) How would you define your role as part of the Urban Systemic Initiative in : 
your district? Probe: How do you support the reform agenda in schools? 






Id) 


How does the district identify teachers’ instructional needs? 

• Please provide examples of your efforts in this area. 






le) 


How does the district support the identified instructional needs of teachers? 
• How do you measure the effectiveness of these efforts? 








1 c) How does the district identify unmet student needs? 

• Please provide examples of your efforts in this area. 

Id) How does the district support these unmet student needs? 

• How do you measure the effectiveness of these efforts? 








le) How does the district identify the needs of administrators? 

• How does the district support the identified needs of administrators? 




Curr 


2 


How would you define standards -based instruction? 


1 1 




2a) 


How is standards-based instruction implemented in the math and science classrooms in 
your district? 






2b) 


How have you integrated standards-based instruction into the mathematics and science 
curriculum? 






2c); 


How do you determine the effectiveness of standards-based instruction? 






2d) 1 


How has the USI affected Student Achievement in mathematics? 






2e) 


How has the USI affected Student Achievement in science? 






20 


How has USI affected technology in mathematics? 






2g) 


How has USI affected technology in science? 




All 


3 


Please describe any major changes that have occurred in your USI plans since its 
implementation. 






3a) 


Why were these changes made? 






3b) 


What new or different organizational structures or patterns have been created as a result of 
the USI? 

Please provide examples 




All 


4 1 What policies (at the national, state, and district levels) regarding math and/or 

1 science instruction/achievement have the biggest impact on your district? 


2 1 




4a) j What do you consider to be your role in implementing national and state policies in your 
1 district? 






4b) 1 What policies support or hinder the implementation of the USI in your district? 






4c) \ What national, state, or local politics influenced decisions concerning your USI? 
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Questions 


Driver 1 




4d) 


How have NSF's USI drivers influenced local policy? (include a handout summary of 
drivers) 






4e) 


Has the NSF USI influenced policy-making in other areas? 


III 




40 


What kind of working relationship do you feel that the district has with the NSF in 
administering the USI? 






- ^g) 


Has that relationship changed during the course of the grant? 






5 


What social equity issues or problems are particularly troublesome in your district? 


2 1 




5a) 


What policies or procedures have been implemented to reduce social these inequities? 


ft 




5b 


How has the USI has been useful in addressing such issues/problems? 


1 




6 


What is the district's policy for evaluating school based administrators? 


3, 7 




6a) 


What measures, criteria, or processes are used to assess administrative performance, 
formally and informally? 






6b) 


How do you determine the effectiveness of these evaluation procedures on student 
achievement in mathematics and science? 






6c) 


How are the procedures for evaluating district-based administrators different from school- 
based administrators? 


1 




6d) 


How has district policy addressed improving the climate ofthe working environment at the 
district level, the school level, and the classroom level? 


1 




7 


During the last three years have you or your staff attended any professional 
development activities? 


' 1 




7a) 


When during the year? 

What topics or programs were covered? 

Who sponsored the professional development programs.? 






7b) 


What are the percentages of staff who attended? 

Is attendance required? 

For staff who do not attend, how is information disseminated? 






7c) 


In what manner does the district support teachers and administrators to earn advanced 
degrees. 






7d) 


What process did you use in developing the agenda for professional development 
programs? 


1 




7e) 


How is staff development designed, as a series of events or as part of sustained 
engagement? 






70 


What is the role of standards-based instruction and constructivist teaching practices in 
professional development program? 

How was that role established? 






7g) 


How does staff development promote changes in attitudes and instrumentation? 


I 




8 


What are the current policies regarding out-of-field teachers in your district? 


3 II 




8a) 


What steps has the district taken to reduce the number of out-of-field teachers in 
mathematics and science? 






8b) 


What steps have been taken to improve the mathematics and/or science content 
backgrounds of elementary teachers? 




Curr 


9 


How often are district meetings held to discuss the philosophical basis of instructional 
practices, and other issues related to the USI? 

(Frequency, % of attendees, topics). 






10 


How has participation in the USI enabled you to increase the amount of available 
resources 

(technology, instructional materials, funding, time). 

Provide examples. 






lOa) 


What policies were developed or implemented handle the problem of having USI funds 
supplant rather than supplement district support for mathematics and science programs? 






I Ob) 


How were USI funds used to supplement district contributions? Please describe examples. 






I Ob) 


Were there instances were USI funds supplanted normal district support for math and 
science programs? 

Why did this happen? 






11 


What steps have you taken to ensure the sustainability of the USI reform effort? 


3 II 
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12 


Thinking about community resources such as university partnerships, businesses, 
parents, etc., how has their participation in UStrelated or other reform efforts 
impacted your district? 

(What are some other reform efforts?) 


4 




12a) 


Please rate their participation as it impacts the science and/or math programs in your 
district. 






13 


How do you know if students are benefiting from your district’s involvement in the 
USI? 


5,6 




13a) 


What are the particular markers of student achievement/accomplishment for mathematics 
and/or science that you consider important? 
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The decision to use focus groups was made by the research team when the schools and 
teachers were selected to participate in the research study in the fall of 1999. The focus 
groups were organized to inform the teachers about the research project, answer any 
questions, and obtain the opinions of the teachers about professional development 
opportunities they had experienced. The focus group facilitators asked the participants 
open-ended questions about which of their professional development experiences had 
provided the most useful or least useful information. Researchers also asked about what 
teachers wished to have changed about the professional development offered through the 
USI or district. 



For What Study? 

The focus groups were conducted in the 47 schools in fall 1999. The five teachers 
participating in the research and, occasionally, administrators and other teachers 
participated in the focus groups. The focus groups were conducted to inform the Policy 
Study by identifying how professional development policies at the local level have 
impacted teachers’ classroom practices. 



Relationship to Other Parts of the Study 

The teacher focus groups about professional development lent information about Driver 3 
(Unified Application of Resources) in the Six- Driver model. One of the indicators for 
Driver 3 is professional development activities for teachers. In their responses to open- 
ended questions, teachers shared their experiences with professional development 
activities and identified the types that had the most impact on their instructional practices. 
The first Driver in NSF’s model calls for the “implementation of standards-based 
curriculum and/or instmctional materials that are aligned with instmetion and 
assessment.” Professional development policies are the primary vehicle used for 
accomplishing the goals of the first driver. 



Development Process 

The focus group questions were developed during fall 1999, before the first site visits to 
the participating schools. The questions were created to gain an initial understanding of 
the types of professional development activities in which the teachers have been 
involved. 
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Protocol for Teacher Focus Group 



1 . Introduce the teachers to the NSF grant, and ask if they have any questions, 

2 . Professiona 1 development questions : 

Ql: We’d like to you reflect on your experiences with staff development over the last 
four years for mathematics and science. 

Q2: Which professional development experiences were the most useful or helpful for 
you in your classroom instmction? 

Q3: Which professional development experiences were the least useful or helpful for 
you in your classroom instmction? 
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Pul’pose 

The teacher questionnaire was used in conjunction with the teacher focus group protocol 
to solicit individual teacher’s points- of- view. The open-ended nature of the questionnaire 
allowed each teacher an opportunity to describe his or her personal views regarding 
professional development. 



For What Study? 

The teacher questionnaire was given to all participating teachers in Chicago, El Paso, 
Memphis, and Miami in the fall of 1999, as part of the data collection for the Policy 
Study. In all, 148 teachers responded to the questionnaire. 



Relationship to Other Pans of the Study 

The teacher questionnaire informed NSF’s Six- Driver model by providing information on 
several of the Drivers from the teacher’s perspective: Driver 1 (Standards-based 
Instruction), Driver 3 (Unified Application of Resources), and Driver 4 (Mobilization of 
Stakeholders). The professional development questions (Driver 3) included in the 
questionnaire were used with the teacher focus group data to understand the teacher’s 
professional development activities since the beginning of the Urban Systemic Initiative 
(USI) reform efforts in each of the participating cities. 



Development Process 

The items for the teacher questionnaire were developed by the research team in the fall of 
1999 to address how the USI reform has affected the teachers since the inception of the 
reform. The questions touched upon a number of the Drivers, as mentioned above, which 
aided the researchers in developing the monthly correspondence questions to the teachers. 
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Teacher Questionnaire 




1 . 



2 . 



3. 



4. 



5. 



6 . 



7. 



8 . 



What has been your experience with the District’s Urban Systemic Initiative? 

What are your impressions regarding the goals of the District’s Urban Systemic 
Initiative? 

Have you made any changes in your instmctional strategies since the 
implementation of the District’s Urban Systemic Initiative? If so, please describe 
these changes. 

What types of assessment strategies (test, oral reports, projects) do you use? 

Please give some specific examples. How did you choose the methods that you 
use? 

What types of professional development activities (at your school or elsewhere) 
have you participated in through the District’s Urban Systemic Initiative? Please 
comment on the content and format of these conferences/workshops. 

Please comment on the effectiveness of these professional development activities. 
What aspects have you found useful or not useful? What types of professional 
development activities would you like to see in the future? 

What instmctional materials (if any) have been made available to you by the 
District’s Urban Systemic Initiative? Please comment on the effectiveness of 
these materials and describe what would be effective in the future. 



What type of input have you received regarding curriculum and instmction from 
the following: 

a) Parents? 

b) Principal? 

c) Other teachers? 

d) School Board? 

e) Superintendent? 

f) University Partnerships? 

g) Local Businesses? 



In what ways do you utilize 
(individual or organizations) 



community resources and/or school partners 
in your classroom? 
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Purpose 

The correspondence provided information on how policies, student assessments, classroom 
assessments, technology, and curriculum decision-making are viewed by teachers. Teachers were 
also able to articulate how these same issues impact their classroom practices. 

For What Study? 

The teacher survey correspondence was used to collect teacher-reported data for the Mathematics 
and Science Attainment Study and the Policy Study. The correspondence was distributed on a 
monthly basis throughout spring 2000 to teachers participating in the overall study. 

Relationship to Other Parts of the Study 

The correspondence topics sought to obtain information about Drivers 1, Standards -based 
Instruction, and Driver 2, Unified Application of Policy in the Six-Driver model. 



Rel 


ationshi p of Items to IS 


SF"s Six Drivers 




Instrument 


Sample 


Driver 1 


Driver 2 


Driver 

3 


Driver 

4 


Driver 

5 


Driver 

6 


Proposed 
Driver 7 


E-Mail 
Survey 
Topic One 


Teachers 




1,2,3 












E-Mail 
Survey 
Topic 2 


Teachers 


1,2,3, 5 


4, 5 










4 


E-Mail 
Survey 
Topic 2A 


Teachers 


1,2,3,4,5 















Development Process 

The correspondence topics were developed by the research team beginning in fall 1999 and 
administered throughout the school year to establish an on-going communication with teachers 
concerning their views on the implementation of the USl, The topics addressed critical aspects 
of the NSF Drivers: policies, student assessment, classroom assessment, technology, and 
curriculum decision-making and sought to determine the impact of these on teaching practices. 
These topics were selected from the teacher questionnaire (see Teacher Questionnaire 
description) to gather more in-depth information on the topics from the teachers. The questions 
were open-ended and designed to also provide rating data using a zero to three scale. The first 
question on each correspondence included three parts: Part 1 asked teachers to identify 
important aspects about each topic; Part 2 asked teachers to rate the topics impact on 
instructional practices; and Part 3 asked teachers to describe the topics use in their classroom. 
The first question was followed by three open-ended questions asking respondents to elaborate 
and evaluate the impact of each topic on their classroom practices. Email accounts were set up 
so that teachers could respond using the Internet. There were challenges to this method, so 
teachers faxed their responses to the research team instead. 
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Teacher Correspondence #1 
January 2000 

TOPIC 1 : What mandates and policies (that is, the measures you are held responsible for implementing) 
most affect your classroom instruction in mathematics and science? 



1) Please provide a brief description (exact title not required) of each policy and/or mandate. 
Indicate whether you perceive it to be a national, state, district, or school policy/mandate. 

2) Indicate the extent of its effect on your instructional practices. 



0= None 



=Minimal 



2=Moderate 3=Major 



What is your professional evaluation of each policy and/or mandate with respect to its impact on student 
academic success? 



Note: We have provided space for several policies/mandates, but you need not use it all. Provide 
information for those policies/mandates you feel most impact your classroom instruction in mathematics 
and science 



Policy/Mandate 


0-3 


Evaluation of Policy/Mandate 
Impact on Student Academic Success 









Teacher Correspondence #2 
February 2000 

TOPIC #2: How does assessment affect your classroom instructional practices in mathematics and 
science? 



1) Please provide the names of state, district, school, and classroom level assessment instruments 
(exams, tests, portfolios, etc.) for students in your school in general and your class in particular. 
(Column 1) 

2) Indicate the extent of impact of each on your instructional practices. (Column 2) 

0= None l=Minimal 2=Moderate 3=Major 

3) When and how do you prepare students for them? How mu ch time does this preparation entail? 
(Column 3) 



Question #1: Name or 
Description 


Question #2: (0-3) 


Question #3: Preparation Description 









4) What particular issues concerning assessment has your faculty talked about and how did you 
resolve them? 

5) As a school, what has been done to address the assessment needs of the student body? 
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Teacher Correspondence #3 
March 2000 

TOPIC #3: How have your assessment and grading procedures in mathematics and science been affected 
by the USl? 



1 . In column 1 , please name the categories that you use for determining student grades in 
mathematics and science. These might include Tests, Quizzes, Projects, Presentations, Labs, 
Portfolios, Student Self- and Peer Evaluations, etc. 

2. In column 2, please describe the role of each category in your assessment process; e.g. why it is 
included, how often it is used, when it is used, whether it “counts” every time, etc. 

3. In column 3, indicate the effect of professional development on the inclusion of those categories: 

0=None l=Minimal 2=Moderate 3=Major 



Question #1: Name 


Question #2: Category Description 


Question #3: 
(0-3) 









4. Please describe any professional development that affected your decisions to include particular 

categories in your assessment and grading process. 



5. Please describe the processes by which you determine cumulative grades for particular 
time periods including 

^ end of marking periods end of semesters 

end of years others particular to your school 

The processes might include weighted or unweighted formulas, rubrics, portfolio, district guidelines 
etc. 

6. Please describe any changes in your assessment and grading procedures in mathematics and science 
over the past five years. 

For one math or science class, please provide a copy of your grade book pages from the beginning of the 
1999-2000 school year through the end of the most recent grading period. In the place of student names, 
which should be blocked out , please provide race and gender identifiers. 



Teacher Correspondence #4 
April 2000 

TOPIC #4: How does technology affect your classroom instructional practices in mathematics and 
science? 



1) Please identify the forms and instruments of technology that you and your students use in the 
classroom. (Column 1) 

2) Indicate the extent of impact of each on your instructional practices. (Column 2) 

0= None l=Minimal 2=Moderate 3=Major 

3) When and how are they used? (Column 3) 



Question #1: Name or 
Description 


Question#!: (0-3) 


Question #3: Description of Use 









4) How available is technology to both teachers and students in your school? 
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5) Are there provisions in place to assure equal access to technology for all students and teachers? If so, 
how effective are those provisions? 

6) How have your instructional practices changed as a result of technology? 




1) Please identify the various people that make decisions about what, when, and how you teach. 



2) Indicate the extent of impact of each on your instructional practices. 

0= None l=Minimal 2=Moderate 3=Major 

3) Describe the impact of each decision on your instructional practices. 



Question #1: Name or Title 


Question #2: (0-3) 


Question #3: Description of Impact 









4) Describe your school’s formal and informal decision-making processes for curriculum and 

instruction - that is, the ways in which you and your colleagues decide what to teach, how to 
teach, and when to teach it. 



5) How would you evaluate the decision-making process for curriculum and 
instruction in your school? In your district? 
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Purpose 

This protocol was created for the Pilot Study of the Community Context (January - June 2000). 
Its initial use was in Chicago with the Chicago Systemic Initiative (CSI) Community Study 
Interview for Identified Partners or Stakeholders. The instrument was used to field test an 
interview protocol and gather information about the mobilization of stakeholders in each of the 
four sites. The purpose of the pilot study was to: gather information about stakeholders and 
partners at two high schools in each site, elicit the multiple definitions of “community” and 
“partners”, and identify resources and stakeholders for participation in the full community study 
conducted from September 2000 to March 2001. 



For What Study ? 

The original instrument was developed to support the pilot study of the community context. 
Researchers in all four sites adapted portions of this instrument for the studies conducted between 
January and June 2000. 



Relationship to Other Pans of the Study : 

Feedback from pilot study researchers helped in the design of the three protocols used in the full 
Study of the Community Context (2000-2001). 



Development Process 

The protocol design came about after group discussions during the Technical Advisory Network 
meeting in February 2000 at the University of South Florida. An evaluation of the items and 
discussion of the instrument’s utility for the full study occurred during a meeting of all 
researchers on September 17, 2000. All researchers present collaborated in the identification of 
questions that would be most appropriate for the instruments in the full study. 
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Pilot Study Interview Protocol 

CSI Community Study Interview for Identified Partners or Stakeholders (spring 2000') 



Background 

Can you provide a little background information for me: 

How long have you been in ? What is your role/position in ? 

How did you come to occupy this role/position? 

Can you tell me what it is that your position consists of (i.e., what are your responsibilities)? 

Stakeholder Questions 

1. How has the school has changed since the recent reform? 

2. How would you define the boundaries of the school's "community"? 

Rationale: There are multiple definitions of community, from the district to the student level. It is 
important to elicit as many perceptions of this term as possible to accurately understand the 
relationships between schools and their overlapping “communities.” 

3. How would you say that the community has changed since the recent reform? 

Rationale: If respondents perceive the “community” to be the school, the students, or the district, 
they may be able to talk about how increased support for math and science has affected this 
community. 

4. Are the changes in the school and the changes in the community connected to one another, and if so, 

how? 

5. Can you describe any local organizations agencies, businesses, universities, parent groups, or persons 

that engage with the school to assist the school in its efforts of reform? (e.g., these might include 
community organizations, local politicians, clergy, parents, community activists, or other external 
partners). 

Rationale: The “emic” identification of stakeholders is a more accurate description of the 
individuals and organizations that support math and science (or education in general). Individuals 
may be aware of relationships that have not been identified in archival research, document review, 
etc. 

6. Canyou tell me a little about how became involved with the high school? (i.e., how the 

relationship came about, how long has this relationship existed, what does the relationship or 
partnership consist of, etc) 

7. Did your school have connections to any of these groups prior to the reform and if so, can you describe 

any changes that may have occurred in this relationship since reform? 

8. Can you describe any resources or assets these groups or persons provide? (e.g., money, materials, time, 

staff or expertise, relationships)? n addition to their connection with the school, are any of these 
groups connected to one another? 

9. What would you say are the strengths of these relationships (please distinguish between the different 

partnerships that you are describing - i.e., if one partnership has a particular strength but another 
does not, please make this distinction for us)? 

10. What would you say are the difficulties associated with these relationships or partnerships (if any)? 

1 1. How would you say that the local (school) community relates to the school (e.g., perceptions, activities 

and involvement with either the school or with the partnerships related to the school)? 

12. What do you perceive to be some of the continuing needs of the high school? 

13. In what direction do you see the school going in the near future? In the distant future? How do you see 

the involvement of these various “stakeholders” in this projection? 

14. Finally, can you offer us some names of persons or groups with whom we might speak regarding the 
partnerships we’ve just been discussing? 
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Pun^ose 

The instruments for the Study of the Community Context consisted of three protocols, one for 
school level staff, the second for parent organizations, school governance, district staff, corporate 
and community partners, and the last for parents of students. They were created to for several 
purposes: facilitate semi-structured interviews to gather information about the mobilization of 
stakeholders in each of the four sites; gather information about stakeholders and partners at two 
high schools in each site; help researchers understand the multiple definitions of “community” 
and “partners;” and to secure measures of importance and effectiveness of partnerships and 
stakeholder relationships. 

For What Study ? 

The Study of the Community Context is part of the Policy Study to investigate the mobilization 
of stakeholders (Driver 4) in the implementation of the Urban Systemic Initiative. 

Relationship to Other Parts of the Study : 

Questions on the principal interview and district interview protocols were used in the design of 
the community study protocols in an attempt to collect similar information from multiple sources. 
For example, question 11 in the principal interview was repeated in the district interview 
(question 12) and used in the community study protocols. The questions were adapted to gather 
more specific information about the roles and responsibilities of the respondents. 



Relationship of items to NSF”s Six Drivers 



Instrument 


Sample 


Driver 

1 


Driver 2 


Driver 3 


Driver 4 


Driver 5 


Driver 6 


Proposed 
Driver 7 


Protocol # 1 


Stakeholders - 
School site 








IB, 1C, 
3B, 3C 








Protocol U2 


Stakeholders 

Governance, 

community, 

district 








4A, 4B 









Devcbpment Process 

The protocol designs were discussed during the meeting of pilot study researchers on September 
17, 2000. After review by research team members, the designs were finalized for the Chicago 
wave of research (September-October 2000). The statement of informed consent and space for 
respondent’s signature were inserted at the top of each protocol to ensure compliance with 
consent requirements. The protocols were changed to eliminate the information about birth date 
(unnecessary information for these informants and challenged by three interviewees) and to add 
space for respondents to identify and rank the importance and effectiveness of stakeholders. The 
adapted instruments were used to collect data during the remaining three site visits (Memphis - 
October-November 2000, Miami - January-February 2001, and El Paso - February-March 2001). 

Coding and Analysis 

A coding rationale and NUD*IST coding tree have been developed by an anthropologist, C. 

Kelly, who served as a consultant with the project during the pilot study. These categories form 
the framework for coding and analysis of the information from this study. All field notes, 
impromptu (non-taped) interviews, formal (taped) interviews, and other narrative information will 
be entered into NUD*IST in order to identify key themes and information about stakeholder 
mobilization at each site. 
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Protocols for Stakeholder Interviews 



INTERVIEW PROTOCOL #1 

School - level staff 

(Partnership coordinators, school-to-work staff, counselors, volunteers, teachers) 



Informed Consent: The purpose of this research has been explained to me. I understand there are no risks involved in my 

participation and that my identity (and other personal information) will be kept confidential by research staff. 

Name; Signature: 

Position: 

Gender; F M 



Race/Ethnicity; 



White(non-Hispanic) 

Asian/Pacific Islander 



Hispanic_ 



African American_ 
Native American 



Other 



How long have you been in (this organization)? 

What is your role/position in (this organization)? 

How did you come to occupy this role/position? 

Can you tell me what it is that your position consists of (i.e., what are your responsibilities)? 
STAKEHOLDER QUESTIONS 

1 . A. In what ways do you think partnerships and stakeholder involvement in the school h ave changed 

since the recent USI reform (1994 - 2000)? 

B. Can you rank the importance of stakeholder involvement to students’ achievement and attainment on 
a scale of 0 ~ 3? Please explain your ranking 



0 1 


2 


3 


Not important 




Extremely Important 



C. Can you rank the effectiveness of stakeholder involvement to students’ achievement and attainment 
on a scale of 0 - 3? Please explain your ranking. 



0 

Not effective 



3 

Extremely effective 



3. 

4. 



Definitions and perceptions of “community” 

A. How would you define the boundaries of the school's "community?” (Probe for specifics about 
neighborhoods or the community surrounding the school) 

B. Would students and parents use the same definition(s)? How might they differ? 

How does the local community or neighborhood relate to the school (e.g., perceptions, activities and 
involvement with either the school or with the partnerships related to the school)? 

A. Can you identify any “partners” that engage with the school to assist the school in its efforts of 
reform? Please rank them in terms of their importance for improving students’ math and science 
achievement and attainment (0 - n ot important to 3 - extremely important) 



0 1 2 


3 


Not important 


Extremely Important 


B. Please rank the effectiveness of these “partners” 


in contributing to the improvement of math and 


science achievement and attainment (0 - not effective to 3 - extremely effective) 


0 1 2 


3 


Not effective 


Extremely effective 



Type of organization 


Name 


Nature of partnership 


Ranking 


Local organizations 








Agencies 








Businesses 








Universities 








Parent groups 








Individuals (politicians, 
clergy, parents, 

community activists) 








Community organizations 
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INTERVIEW PROTOCOL #2 

PTA/PTO, SAC, LSC Members 

District level Staff/Corporate and Community Partners 

1, How long have you been involved with (this organization)? 

How are you involved? 

What prompted your involvement? 

Do you hold a specific position within (this organization)? 

Are there specific terms (length of participation) associated with your position? 

2. Are you aware of the reform efforts at the school? (PROBE FOR KNOWLEDGE OF USI. Use names of 
programs or activities funded by USI if respondent is not familiar with USI specifically.) 

3. Definitions and perceptions of “community” 

How would you define the boundaries of the school's "community"? 

Would students and parents use the same definition(s)? How might they differ? 

Do you think the math and science reform has affected the community (Probe: either the school’s 
“community” or the community-at-large)? If yes, how. 

STAKEHOLDER INFORMATION 

4. A. Can you identify any “partners” that engage with the school to assist the school in its efforts of reform? 
Please rank them in terms of their importance for improving students’ math and science achievement and 
attainment (0 - not important to 3 - extremely important) 

B. Please rank the effectiveness of these “partners” in contributing to the improvement ofmath and science 
achievement and attainment (0 - not effective to 3 - extremely effective) 



0 1 


2 


3 


Not effective 




Extremely effective 



Type of organization 


Name 


Nature of partnership 


Ranking 


Local organizations 








Agencies 








Businesses 








Universities 








Parent groups 








Individuals (politicians, 
clergy, parents, 

community ' activists, 
external partners) 








Community organizations 









INTERVIEW PROTOCOL #3 

Parents of Students (Student Engagement Study, school staff, volunteers, etc.) 

1 . What do you know about the Urban Systemic Initiative for math and science reform at your child’s school? 
(Probe with names of specific programs or activities funded by USI). 

2. How much do you know about the reforms in math and science at your child’s school? 

3. Have you seen any changes in your child’s math and science instruction since he or she started high school? 

4. Is the homework different from the homework (schoolwork) that you did in high school? If so, how? 

5. If you have older children, is the homework (schoolwork) different from the homework the older children did 
in high school? If so, how? 

6. Do your child’s classes seem “different” than classes were five years ago? PROBE 

7. What does your child’s school tell you about what is going on, specifically math and science instruction or 
reform? How do you get the information (flyers, brochures, calls, letters, other)? 

8. How does your child’s school help you better understand the math and science reforms? PROBE 

Events like math fairs or science fairs 
Workshops on homework assistance or 
Workshops on understanding the reform programs; 

Trainings for parents in the new instruction and course content 
Family math or Family science nights at the school 

9. Have you attended any of these? 

10. Do you think that these kinds of supports need to be available to parents? 
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12. 

13. 

14. 

15. 



16. 



17. 



18. 



A. What do you know about the math and science coursework needed for high school graduation or college? 
B. How important to you is your child taking algebra, geometry, and the science courses necessary for 
admission to college? Rank from 0 - not important to 3 - extremely important. Please explain your | 
ranking. 

What does your child’s math and/or science teacher tell you about math and science instruction or reform? | 
How do you get the information (conferences, meetings with the teacher, flyers, calls, letters, other)? 
What classes are your child taking? (PROBE - IB, AP, etc.) 

Do you help your child to choose his/her class schedule? 

Please tell us your impression ofyour child’s experience with math instruction. PROBE 
What do you hear him/her saying about the class? 

The class content? 

Do you see your child using his/her math and science knowledge in activities or projects around the I 
house? 

Please tell us your impression ofyour child’s experience with science instruction. PROBE 
What do you hear him/her saying about the class? 

The class content? 

Do you see your child using his/her math and science knowledge in activities or projects around the I 
house? 

Testing - tell us what you think about the state testing being done in schools today (TCAP, TAAS, FCAT, | 
CASE). 

Do you help your child prepare for these tests? If so, how? 

Do you understand how these tests and scores are used to guide your child’s school program? I 
What resources in your neighborhood are available to help your child improve/strengthen his or her math or | 
science performance? 
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